Papers with pretrained masked language model
Pretrained Bidirectional Distillation for Machine Translation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing studies have focused on language knowledge transfer from pretrained models to neural machine translation models. |
| Approach: | They propose to use masked language pretraining to efficiently transfer bidirectional language knowledge to NMT models. |
| Outcome: | The proposed method can significantly improve machine translation performance and achieve competitive or even better results than previous methods. |
Swap and Predict – Predicting the Semantic Changes in Words across Corpora by Context Swapping (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Detecting semantic changes of words is an important task for various NLP applications that must make time-sensitive predictions. |
| Approach: | They propose a method that randomly swaps contexts between two different corpora to detect whether a given word changes its meaning . they then use a pretrained masked language model to generate contextualised word embeddings of w, which are then used to predict the semantic changes of words in four languages . |
| Outcome: | The proposed method achieves significant performance improvements compared to baselines for the English semantic change prediction task. |
CxLM: A Construction and Context-aware Language Model (2022.lrec-1)
Copied to clipboard
| Challenge: | Constructions are direct form-meaning pairs with possible schematic slots . however, these slots are constrained by the embedded construction and the context . we propose that a conditional probability distribution could be described but language models cannot capture this distribution. |
| Approach: | They propose that a conditional probability distribution could describe constructions’ schematic slots. |
| Outcome: | The proposed model predicts masked slots more accurately than baselines and generates structurally and semantically plausible word samples. |
IndiGEC: Multilingual Grammar Error Correction for Low-Resource Indian Languages (2025.emnlp-main)
Copied to clipboard
| Challenge: | despite growing interest in GEC, most research has focused on English due to the lack of benchmark datasets for low-resource lan-guages. |
| Approach: | They propose a new approach to generate high-quality synthetic data for GEC using monolingual corpora. |
| Outcome: | The proposed framework outperforms other monolingual methods in English, Hindi, Bengali, Marathi, and Tamil. |